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Breakthrough Propulsion Physics Project: 
Project Management Methods 

Marc G. Millis 

National Aeronautics and Space Administration 
Glenn Research Center 
Cleveland, Ohio 44135 


Abstract 

To leap past the limitations of existing propulsion, the NASA Breakthrough Propulsion Physics 
(BPP) Project seeks further advancements in physics from which new propulsion methods can eventually 
be derived. Three visionary breakthroughs are sought: (1) propulsion that requires no propellant, (2) 
propulsion that circumvents existing speed limits, and (3) breakthrough methods of energy production to 
power such devices. Because these propulsion goals are presumably far from fruition, a special emphasis 
is to identify credible research that will make measurable progress toward these goals in the near-term. 
The management techniques to address this challenge are presented, with a special emphasis on the 
process used to review, prioritize, and select research tasks. This selection process includes these key 
features: (a) research tasks are constrained to only address the immediate unknowns, curious effects or 
critical issues, (b) reliability of assertions is more important than the implications of the assertions, which 
includes the practice where the reviewers judge credibility rather than feasibility, and (c) total scores are 
obtained by multiplying the criteria scores rather than by adding. Lessons learned and revisions planned 
are discussed. 


Introduction 

The NASA Breakthrough Propulsion Physics (BPP) Project was funded from 1996 through 2002 to 
assess the prospects from emerging science that might lead to breakthrough methods of spaceflight. Since 
this objective is more visionary than typical propulsion research, special attention is given to how such a 
Project is managed. Balancing credibility and vision evokes special challenges, as does the requirement to 
produce near-term progress toward goals that are beyond foreseeable fruition. The management 
challenges and resulting methods are described. 


Project Goals 

As its name suggests, the Breakthrough Propulsion Physics Project is specifically looking for 
propulsion breakthroughs from physics. It is not looking for further technological refinements of existing 
propulsion methods. Such refinements are being explored in other NASA projects (2003 NASA Strategic 
Plan, Bachtel 1997). This Project seeks further advances in science from which genuinely new technology 
can eventually be derived - technology to suipass the limits of existing methods (Millis 1999a). 

The research is focused toward the following three specific breakthroughs — the “Grand Challenges” 
that could revolutionize spaceflight and enable interstellar missions: 

Grand Challenge 1: Mass . — Discover new propulsion methods that eliminate or dramatically reduce 
the need for propellant mass. This implies discovering fundamentally new ways to create motion, 
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presumably by interaction with the properties of space, or by the interaction of matter, energy, and space- 
time, including the possibility of manipulating gravity or inertia. 

Grand Challenge 2: Speed . — Discover how to circumvent existing limits to dramatically reduce 
transit times. This implies discovering a means to move a vehicle at or near the actual maximum speed 
limit for motion through space, or by some other means, such as by interaction with space-time itself to 
circumvent normal limits. 

Grand Challenge 3: Energy . — Discover fundamentally new modes of onboard energy generation to 
power these propulsion devices. This third goal is included since the first two breakthroughs could require 
breakthroughs in energy generation, and since the physics underlying the propulsion goals is closely 
linked to energy physics. 

The physics underlying these ambitions includes experiments and theories regarding the coupling of 
gravity, electromagnetism and space-time, properties of the space vacuum and inertial frames, quantum 
level effects, waip drives, wormholes, and anomalous effects of force-production or energy-exchange. 


Project Implementation 

This Project was invited as part of a 1996 reassessment of NASA propulsion research efforts (Bachtel 
1997 and Millis 1999a), yet its implementation was proposed as a sequence of conditional steps, where 
continuation was contingent upon successful completion of the prerequisite steps (Millis 2000). This 
conditional posture was proposed to ensure that there were enough foundations for launching into, and 
sustaining, such a visionary project. These conditional milestones and the documentation summarizing 
their findings (cited in parentheses) are as follows: 

1996 — Determined that sufficient scientific foundations exist (Millis 1996). 

1997 — Determined that affordable research candidates exist (Millis 1999b). 

1998 — Devised means to prioritize and select research tasks (Millis 1999a). 

1999/2000 — Solicited and selected 1st round of research tasks (NASA Solicitation 

NRA-99-LeRC-l). 

In October 2002, the sponsoring Program, the Advanced Space Transportation Program, was 
reorganized. All research was deleted that was at less than Technology Readiness Level 3 (conceptual 
design tested analytically or experimentally), including the BPP Project. In November 2002, the Final 
Report of Aerospace Commission, whose task was to assess the health of the Nation's aerospace industry, 
recommended: “ . . .that the federal government significantly increase its investment in basic aerospace 
research . . .” (Walker 2002). More specifically, in Chapter 9, the Commission recommended: “In the 
longer-term, breakthrough energy sources that go beyond our current understanding of physical laws. . . 
must be credibly investigated in order for us to practically pursue human exploration of the solar system 
and beyond. These energy sources should be the topic of a focused basic research effort.” At this time it is 
uncertain if support for the BPP Project will resume. 

A recent compilation of the findings of 16 individual research tasks, from this Project and several 
parallel programs, indicates that about a third of the approaches were found not to be viable, a quarter 
have clear opportunities for sequels, and the rest remain unresolved. It is expected that new approaches 
will continue to emerge (Millis 2004). 


This Report 

This report is intended for two audiences; mangers of other cutting-edge projects, and researchers 
interested in breakthrough propulsion physics. The management methods are offered as a guide for others 
that face similar challenges. Many of the methods presented here can be adapted to other long-range 
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research projects. Also, researchers can benefit from examining the research selection criteria, to improve 
their future proposals, should funding resume for research solicitations. 

The special challenges of visionary research are outlined, as are the mitigation strategies for dealing 
with these challenges. A special emphasis is placed on the process to review, rank, and select research 
options. The lessons learned over the course of the Project are also offered, as well as the planned changes 
that respond to the lessons learned. 


Special Challenges and Mitigations 

A normal challenge of any research project is to prioritize tasks to efficiently direct resources to the 
best prospects. In addition to this typical challenge, the BPP Project faces the challenge of making 
credible progress toward the incredible ambition of breakthrough spaceflight. Because the desired 
propulsion breakthroughs are presumably far from fruition, and perhaps even impossible, specific 
strategies have been devised to mitigate the risks and maximize the progress of such visionary research. 

Exploring the edge of knowledge for profound discoveries evokes special challenges. In addition to 
the normal challenges of scientific research — discovering how nature truly works — the provocative 
character of Grand Challenges can encumber research. First, by pursuing truly profound improvements in 
the human condition, the stakes are higher and accordingly emotions run higher. Second, by operating on 
the genuine edge of knowledge, instead of exploring refinements of established knowledge, controversial 
ideas are encountered. This combination of heightened emotions and controversy can encumber the 
normal, productive discourse of scientific study. Both skeptics and optimists can prematurely reach 
conflicting conclusions and, in their zeal, fail to communicate with the impartiality needed to rigorously 
identify, test, and resolve the real issues. To mitigate these difficulties and to focus on more constructive 
operating practices, the BPP Project employs the following operating strategies, each of which is 
explained in a subsequent paragraph: 

BPP Operating Strategies 

• Reliability — Success is defined as acquiring reliable knowledge, rather than as achieving a 
breakthrough. 

• Immediacy — Research is focused on the immediate unknowns, make-or-break issues, or curious 
effects. 

• Measured — Progress is measured using a combination of the scientific method and applicability. 

• Iterated — Overall progress is achieved by repeating cycles of short-term, incremental tasks. 

• Diversified — Multiple, divergent research topics are explored simultaneously. 

• Impartial — Reviewers judge credibility and relevance, but are not asked to predict feasibility. 

• Empirical — Preference is given to experiments and empirical observations over purely analytical 
studies. 

• Published — Results are published, regardless of outcome. 


Reliability 

Although it is a common practice when advocating research to emphasize the ultimate technical 
benefits, this practice is not constructive on topics as visionary and provocative as BPP. Instead, it is more 
constructive to emphasize the reliability of the information being offered. Compared to other space 
propulsion research, new propulsion physics is at its infancy. It is expected, therefore, that any practical 
embodiment is years, perhaps decades away, if not impossible. Although breakthroughs, by their very 
definition, happen sooner than expected, no breakthrough is genuine until it has been proven to be 
genuine. Hence, the reliability of the information is a paramount prerequisite to the validity of any 
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conclusions. To place the emphasis on where it is needed, the BPP Project does not consider any 
approach unless the credibility criteria are satisfactorily addressed, regardless of the magnitude of claimed 
benefit. Success is defined as acquiring reliable knowledge, rather than as achieving a breakthrough. 


Immediacy 

Another technique, to shift the emphasis away from provocative situations and toward constructive 
practices, is to focus the research on the immediate questions at hand. These immediate unknowns, issues, 
and curious effects can be identified by comparing the established and emerging physics to the BPP 
Grand Challenges. The scope of any research task should ideally be set to the minimum level of effort 
needed to resolve an immediate “go/no-go” decision on a particular approach. This near-term focus for 
long-range research also makes the research tasks more manageable and more affordable. Specifically, 
the Project requests that any proposed research be configured as a task that can reach a reliable conclusion 
in one to three years. Should the results be promising, a sequel can be proposed in the next solicitation 
cycle. 


Measured 

To help guide researchers to identify a suitable research increment, and to provide managers a means 
to measure progress, the Scientific Method has been adapted as a readiness scale in a manner similar to 
how the Technology Readiness Levels are used to measure technological progress (Hord 1985). These 
new readiness levels are detailed in appendix A. Specifically, the readiness scale developed for the BPP 
Project consists of three stages that gauge the applicability of the work (reflecting how research can 
evolve from the more general, to the more specific application), and within each of these 3 stages, 5 steps 
of the scientific method are repeated (from recognizing the problem, through testing the hypothesis). This 
equates to 15 levels of relative maturity, with the most advanced level being equivalent to Technology 
Readiness Level 1 (basic principles observed and reported). Once the maturity of a research objective has 
been identified relative to this scale, the next logical increment of research would propose to advance that 
topic to the next higher level. This is consistent with the incremented research strategy. 


Iterated 

To accumulate progress over the long term, the original intent of the Project was to solicit a suite of 
proposals every two to three years, and to let the lessons learned from the prior suite influence the next 
round of selections. This provides an opportunity for new approaches, sequels to the positive results, and 
redirections around null results. At any point, if a research task leads to the discovery of a new propulsion 
or energy effect, it can be pulled out of this process into its own advancement plan. At the time of this 
writing, however, funding for future BPP solicitation cycles has been deferred. Only one solicitation cycle 
was completed during the funded tenure of the Project. The next cycle was to be implemented as an 
Internet based system, which is discussed in the “Revisions Initiated” section. 


Diversified 

In addition to the techniques already mentioned, the BPP Project aims deliberately to address a 
variety of research approaches in each review cycle. In simple terms, this is to diversify the research 
portfolio. It is far too soon, in the course of BPP, to down-select to just one or two hot topics. This is in 
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contrast to the more common research on advanced propulsion that deals with topics above Technology 
Readiness Level 1 (basic principles observed and reported). In those cases, further advancements are 
sought on technical approaches already under study. Although this more common strategy can produce 
advances on the chosen topics, it faces the risk of overlooking emerging alternative approaches, and faces 
the risk that support will wane unless the chosen topics produce unambiguous positive results. For the 
BPP Project, a more strategic approach was chosen, where cycles of peer-reviewed solicitations examine 
a diverse portfolio of options, and where the decisions build on the lessons learned from the prior cycle of 
research. Furthermore, each of the BPP research options only focuses on an immediate critical issue, to 
systematically chip away at the unknowns toward the overall Project objective. 


Impartial 

In addition to the mandatory emphasis on credibility and the management utility of short-term, 
measured progress, the BPP Project also seeks visionary, breakthrough gains. This presents a special 
challenge, since this invites the kind of controversial ideas that are typically encountered at the edge of 
emerging knowledge. Considering that most historic breakthroughs originally sounded like fringe ideas, it 
is expected that many of the proposed approaches to BPP might sound too visionary at first, or at least 
unfamiliar. It is therefore difficult to sort out the fringe ideas that may one day evolve into tomorrow’s 
breakthroughs from the more numerous, erroneous fringe ideas. Typically, when confronted with such 
unfamiliar ideas, many reviewers will reflexively assume that the idea will not work. Given the kind of 
fundamental investigations sought by this Project, it is difficult to reliably determine such technical 
feasibility during a proposal review. Such an assessment would constitute a full research task itself. 
Instead of judging technical feasibility, proposal reviewers are asked to judge if the task is leading to a 
result that other researchers will consider as a reliable conclusion on which to base future investigations. 
This includes seeking tasks that can reliably demonstrate that certain research approaches are not feasible. 
This posture of judging credibility rather than prejudging correctness is one of the ways that the BPP 
Project is open to visionary concepts while still sustaining credibility. 


Empirical 

Since this NASA Project is interested in advancements that can eventually lead to new technology, 
and since empiricism is necessary to validate theories, there is a decided preference toward empirical 
observations over purely analytical studies — all other factors being equal. Experiments, being hardware, 
are considered closer than theory to becoming technology. Also, experiments are considered a more direct 
indicator of how nature works. Theories are interpretations to explain observations of nature, while the 
empirical data is nature. 


Published 

The final requirement within the BPP strategies is that the research findings are published, regardless 
of outcome. Results, pro or con, set the foundations for guiding the next research directions. Although 
there can be a reluctance to publish null results — where a given approach is found not to work — such 
dissemination will prevent other researchers from following the same dead-ends. In the course of the BPP 
Project, it was learned that more than one organization had independently pursued similar dead-end 
research ideas, because none of these organizations ever published credible results to educate the greater 
community. 
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Evaluation System 


Adaptation of Multiplicative Evaluation Method 

The actual BPP evaluation process was adapted from a procedure developed by Banks of NASA 
Glenn Research Center (appendix B). This procedure starts by assembling a team of representative 
experts and customers of the desired technology. Through brainstorming and voting, the team defines the 
relevant evaluation criteria, and narrows these criteria down to a minimal list with weighting factors for 
each. The group also must distinguish between criteria that are mandatory (criteria that must be met), and 
those criteria that are just enhancing. It is essential that both the customers and the practitioners of the 
research concur with the criteria before applying the criteria to evaluate options. Appendix B offers a 
synopsis of this more general process as a guide to other research Project Managers. 

A recent example of an application of Bank’s process is the selection of the replacement thermal 
control materials for the Hubble Space Telescope (Townsend 1999). The process, as adapted for the BPP 
Project, is detailed next. 

The key features applied to the BPP evaluation process from the Banks procedure are: 

• Multiplicative scoring 

• Familiar scholastic score gradations 

• Independent, minimum criteria, concurred by team 


Multiplicative scoring. — To quickly filter out substandard submissions, it is desired to have a feature 
whereby any failure to meet a mandatory criteria will eliminate the entire submission from competition. 
To provide this feature as an integral part of the evaluation system, the total score is determined by 
multiplying together, rather than by adding , the individual criteria scores. In this manner, any zero score 
(failing grade) on any mandatory criteria will result in a total score of zero. 

To implement such a system, there are 3 details to take into account: (1) how to handle nonmandatory 
criteria, (2) how to handle weighting functions, and (3) how to normalize scores. The sample equation 
below illustrates a multiplicative system for 2 mandatory criteria and 1 nonmandatory criteria: 


Total Score = 


\ a 


\ n a) 


V n bJ 


c+c ■ 

y TL min 

N r 


\c 


where 

• A, B, C... represent criteria scores, 

• a, b, c... are weighting factors, where 1 is the maximum value, and lower priorities are fractions 
of 1 . 

• N a , N b , N c ■■■ are normalizing functions, 

• C ni in is a preset value to prevent the parenthetical term from equaling zero, thereby making criteria 
C nonmandatory. 

To allow nonmandatory criteria into a multiplicative system, two different approaches can be 
employed. The easiest, and the way employed with the BPP process, is to just assign a score range for 
that criteria where the lowest possible score is not zero. The alternate approach, shown in the equation 
above, is to include a nonzero value in the criteria’s equation. This second approach, however, 
complicates how normalizing functions are included. 
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To accommodate weighting factors, exponents are used analogously to the way coefficients are used 
for additive systems. It is recommended to use positive values equal to or less than 1 for these exponents, 
where an exponent of 1 represents the highest priority. 

In practice, the effect of the weighting functions also is tied to the maximum-point-value that each 
criteria can attain. Therefore it is necessary that each criteria be normalized to the same maximum-point- 
value (the terms within the parentheses) prior to applying the weighting exponents. For normalization, 
which means equalizing each criteria prior to applying its weighting function, a simple fractional 
coefficient is applied, so that the maximum possible values of all the criteria are equal. 

Although a generic set of equations can be derived for how to implement a multiplicative system that 
accommodates all possibilities of mandatory and nonmandatory criteria, and accommodates criteria with 
differing scoring ranges, it is far simpler to implement the system with constraints on the scoring ranges. 
If all criteria have the same maximum point value, no normalization is required. If all nonmandatory 
criteria have a nonzero value as their minimum possible score, then no additional constants or associated 
normalization functions are required. 

Scholastic grading . — Experience has shown that an evaluation depends not only on the perceived 
merit of the idea, but also on the evaluators’ interpretations of how to score the idea. For example, if the 
scoring range is 0 to 25 on a given criteria, such as with the Small Business Innovative Research (SBIR) 
evaluations (NASA SBIR homepage: http://sbir.nasa.gov), two different evaluators may use significantly 
different point values to mean the same grade. To avoid this problem, it is recommended to use a familiar 
and limited grading system such as the scholastic 4-point scale: 


A (4 points) 
B (3 points) 
C (2 points) 
D ( 1 point) 
F (0 points) 


Excellent or outstanding, meeting the criteria to the maximum amount. 
Good, or well above average. 

Average, or the score to use if there is no reason to score high or low. 
Poor or well below average. 

Fails to meet the criteria. 


In those cases where these discriminators do not fit, it is still recommended to have the scoring range 
limited to about 5 gradations where possible, and to have clear text explanations to accompany each 
gradation. Since the final scores combine several criteria, it is possible to get sufficient distinctions with 
the total final scores even with such limited gradations. 

Team concurrence on criteria . — As mentioned before, the Banks evaluation process includes a stage 
where a team of representative experts and customers develop and eventually concur on the set of 
evaluation criteria. In the case of the BPP Project, the criteria were evolved over several years and 
amongst a variety of teams. These criteria reflect the BPP strategies described earlier. In the early stages 
of the Project, a “Product Definition Team” helped devise the first set of criteria (Millis 1996). This team 
included a mix of both customers of the research (NASA, DOD, DOE) and practitioners of research 
(physicists from a variety of organizations). A revised set of criteria were tested during a workshop in 
1997 (Millis 1999b), and a further revised set (Millis 1999a) was finally adapted for use in the formal 
solicitation in 1999 and 2000. (NRA-99-LeRC-l). The lessons learned through this process were 
incorporated into the version presented in this report. 


Two-Stage Evaluation Process 

The actual research prioritization and selection process employed by the BPP Project follows a two- 
stage peer-review process. In the first stage, proposals are numerically scored relative to the criteria, and 
in the second stage a diverse suite of proposals are selected from the top-ranking candidates. 

In the first stage of the review process, proposals are subject to scientific review by discipline 
specialists in the area of the proposal. Proposals are reviewed by a combination of in-house and selected 
external reviewers, with due regard for conflict of interest and protection of proposal information. These 
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external reviewers may be from other NASA centers, other Government Labs, Universities, or Industry. 
Each proposal is evaluated by several evaluators, who assign numeric scores for each criteria of the BPP 
Project. By having multiple reviewers per proposal, the standard deviations in their scores can be used as 
an indicator of reviewer agreement. 

To streamline this process, the proposal submitters were asked to encapsulate the key points of their 
proposal onto a summary form. This form was configured to pull together the information that pertained 
most directly to the evaluation criteria, and to present this information in a standard format to make it 
easier for the reviewers to find the key information. An example of the Research Increment Summary 
Form, being drafted for a future solicitation, is presented in appendix C. 

The second stage of the review process involves the compilation and review of these scores by a 
smaller team of reviewers, again with due regard for conflict of interest and protection of proposal 
information. This team examines the top ranking proposals and checks for any disparity in review scores 
on any given proposal. After a suite of top ranking proposals is identified through this secondary review, 
a diversified set of research tasks is selected up to the limit of available funding. 

A suite of different approaches is desired to “diversify the portfolio” since it is still too early to tell 
which research paths will lead most directly to breakthroughs. This means that the proposals selected for 
award may not necessarily be selected contiguously from the highest ranking set. For example, if the top 
two ranking proposals are both to perform an experimental test of a given theory, “A”, and the third- 
ranking proposal is to test a different theory, “B”, then it is the prerogative of this selecting team to award 
the best proposal on “A” and to the third-ranking proposal on “B” while skipping the second-ranking 
proposal on “A”; if this supports diversification of research. 


Selection Criteria 

To evaluate and select research topics for the BPP Project, the criteria described below were 
developed from the methods cited previously. These criteria are a derivative of the criteria used in the first 
BPP research solicitation (NRA-99-LeRC-l). 

Of these, “Technical Relevance” relate directly to the BPP Grand Challenges, the “Credibility” 
criteria judge the reliability of the research, and the “Resource” criteria address affordability and 
timeliness. The total composite score is achieved by multiplying the individual criteria scores as 
illustrated in the equation at the end of this section. This provides the feature whereby a failure to meet 
any mandatory criteria (zero score on criteria) will result in a total score of zero. 

As mentioned earlier, technical feasibility is not being judged with these criteria. Given the kind of 
fundamental investigations sought by this Project, it is difficult to reliably determine technical feasibility 
during a proposal review. Such an assessment would constitute a full research task itself. The burden of 
addressing feasibility, via a discriminating test, is a required part of the research proposal, and adherence 
to this requirement is addressed by criteria 7. Instead of judging technical feasibility, proposal reviewers 
are asked to judge if the task is based upon credible foundations, and leading to a result that other 
researchers will consider as a meaningful and reliable conclusion on which to base future investigations. 
This includes seeking tasks that can reliably demonstrate that certain research approaches are not feasible. 
This posture of judging credibility rather than prejudging correctness, is one of the ways that the BPP 
Project is open to visionary concepts while still sustaining credibility. 

Technical Relevance-. 

1 . Gain — Magnitude of performance improvement relative to all three of the BPP Grand 
Challenges, assuming the approach under consideration ultimately reaches fruition. 

2. Empiricism — Does the topic deal with tangible physical effects or just theory? 

3. Readiness — The present maturity of the topic/concept under study, as measured using the 
Applied Science Readiness Levels (appendix A). 
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4. Progress — Magnitude of progress to be achieved by the research task, as measured by the 
difference in the readiness now (criteria 3), and the anticipated readiness level to be reached upon 
completion of the task, as measured using the Applied Science Readiness Levels (appendix A). 

Credibility. 

5. Foundations — Based on credible references. 

6. Contrasts — Compared to current credible interpretations. 

7. Tests — Leading toward a discriminating test. 

8. Results — Probability that the task will result in knowledge that will be a reliable foundation for 
future decisions. 

Resources : 

9. Triage — Will it be done anyway or is it unique to this Project? 

10. Cost — Funding required (reciprocal scoring factor). 

11. Time — Time required to complete task (reciprocal scoring factor). 

Composite equation. — The total composite score is achieved by multiplying the individual scores, as 
illustrated in the equation below. This has the feature whereby a failure to meet any mandatory criteria 
(zero score on criteria) will result in a total score of zero. 

T otal Score = G WG E WE )' WV i^/p NV )"^ CF WCF CC WCC CT WCT CR WCK TR WTR C wc TI wri 


Criteria 

Variable 

Name 

Score 

Range 

Equation 

Normalizing Variable 

Weighting 

Variable 

1: Gain 

G 

0 to 4 

G = (GM + GS + GE)/ 3 

(1) 

WG 

Gain, goal 1 — Mass 

GM 

0 to 4 

- 

- 

- 

Gain, goal 2 — Speed 

GS 

0 to 4 

- 

- 

- 

Gain, goal 3 — Energy 

GE 

0 to 4 

- 

- 

- 

2: Empiricism 

E 

1 to 4 

- 

- 

WE 

3: Readiness (now) 

RN 

Oto 15 

- 

RNV= 15/4 (2) 

WRN 

4: Progress 

P 


P = RA- RN 

PNV = 1/2 (3) 

WP 

Readiness (after) 

RA 


- 

- 

- 

5: Credible foundations 

CF 

0 to 4 

- 

- 

WCF 

6: Credible contrasts 

CC 

0 to 4 

- 

- 

WCC 

7: Credible tests 

CT 

0 to 4 

- 

- 

WCT 

8: Credible results 

CR 

0 to 4 

- 

- 

WCR 

9: Triage 

TR 

1 to 4 

- 

- 

WTR 

10: Cost (discrete bands) 

C 

0 to 4 

- 

(4) 

WC 

1 1 : Time (discrete bands) 

Tl 

1 to 4 

- 

(4) 

WTI 


Notes: 

1 . The 1/3, embedded in the Total Gain equation, is to normalize the total gain to a maximum value of 4. 

2. The Readiness Normalization Value ( RNV) is set to 15/4 so that a ranking of Technology Readiness Level 1 (score = 15) equates to a score 
of 4. 

3. The Progress Normalization Value ( PNV) is set to 1/2, so that the typical progress of incrementing up one level of the Applied Science 
Readiness Levels (appendix A) equates to a score of 2 (an average condition). 

4. For Cost and Time, the scoring gradations are set to discrete bands rather that directly entering time or cost, and where the higher scores 
reflect lower cost and time. Specific scoring gradations examples are in appendix D, “Research Increment Evaluation Instructions.” 


Criteria explained . — The following text more thoroughly describes each prioritization criteria. To see 
how these apply to a review, including the scoring gradations, refer to appendix D, “Research Increment 
Evaluation Instructions.” 
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Criteria 1: Gain on goals 

This criteria grades how the proposal relates to all three of the BPP Grand Challenges, assuming that 
the concept behind the proposed increment of research ultimately reaches fruition. Each Grand Challenge 
is graded separately and the final Gain criteria involves the sum of these three subset scores. 

It is mandatory that the proposed work seeks advances in science that are in some way relevant to the 
Project’s three propulsion challenges or any critical issues or unknowns related to these goals. The scope 
is limited to further advances in science from which genuinely new technology can eventually emerge — 
to surpass the limits of existing methods — as opposed to further developments of known technology. This 
means that if the proposed work only builds on known technology, then it fails this criteria. 

Criteria 2: Empiricism 

Does the topic deal with tangible physical effects or just theory? Since this NASA Project is 
interested in advancements that can eventually lead to new technology, and since empiricism is necessary 
to validate theories, there is a decided preference toward empirical observations over purely analytical 
studies — all other factors being equal. Experiments, being hardware, are considered closer than theory to 
becoming technology. Also, experiments are considered a more direct indicator of how nature works. 
Theories are interpretations to explain observations of nature, while the empirical data is observed nature. 
The most desired research task is an experiment that is coupled with theory. Experiments that are backed 
by a sound theoretical foundation provide a means to numerically assess the utility and scalability of the 
effects beyond just a single demonstration experiment. The next preference is experimental work by 
itself; for example, to independently test a claimed anomalous effect. After that, the next preference is 
theoretical work by itself. Lowest on this priority scale is work that only involves comparative studies of 
existing approaches or literature searches. 

Criteria 3: Readiness 

This criteria measures the present maturity of the topic/concept under study, as measured against the 
Applied Science Readiness Levels (appendix A). 

Criteria 4\ Progress 

This criteria measures the anticipated progress to be achieved by the research task, calculated by the 
numeric difference between the before and after readiness scores, as measured using the Applied Science 
Readiness Levels. The level before is simply criteria 3, Readiness. The level after is the readiness level 
expected to be achieved upon completion of the research task. Appendix A contains a full description of 
the meaning of these readiness levels. 

Criteria 5: Credible foundations 

This criteria grades how well the proposed work is grounded in credible foundations. The proposed 
work must be based in some way on data or theories that are in the peer-reviewed literature. Note: 
requiring reference citations is one of the techniques to filter out “fringe” submissions. This is a 
mandatory criteria, which means that a failure to meet this criteria (zero score) will result in a total score 
of zero. 

Criteria 6: Credible contrasts 

This criteria grades how well the authors articulate how their proposed work compares to existing 
credible interpretations, relative to the BPP Grand Challenges. This is to ensure that an idea is oriented 
toward the goals of the Project, and to ensure that the authors have done their homework on the existing 
literature. This not only checks for relevance to BPP, but also positions the work to address the next 
criteria of a discriminating test. Note: requiring reference citations is one of the techniques to filter out 
“fringe” submissions. This is a mandatory criteria, which means that a failure to meet this criteria (zero 
score) will result in a total score of zero. 
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Also, recall from the objectives of the BPP Project, that the perceived correctness of the author’s 
alternative interpretations is not being judged with this criteria. Unless there is some obvious error, it is 
considered too difficult to reliably determine such feasibility during a proposal review. Such an 
assessment would constitute a full research task unto itself. The burden of addressing feasibility, via a 
discriminating test, is addressed by criteria 7. For this criteria, it is important that the author demonstrates 
an understanding of the current, credible interpretations that are cited and is able to contrast this prior 
knowledge to the approach they are offering. 

Recall that the BPP Project examines emerging physics in the context of propulsion and power, and 
as such, there is latitude to consider alternative perspectives beyond that from general physics. Even 
though the current credible interpretations have already passed their own rigorous tests, this does not 
imply that such interpretations are a complete or best representation of the actual underlying physics, 
especially in the context of propulsion and power. Conversely, however, if the proposed interpretation has 
already been raised and dismissed in the open literature, then the author must cite these references and 
address the issues raised. 

Criteria 7: Credible tests 

This criteria judges how well the research advances the topic toward a discriminating test. It is 
required that the proposed work be leading toward a discriminating test or actually be a discriminating 
test. If a discriminating test can be completed within the budget and time guidelines requested of 
proposals, it is necessary that the test actually be proposed. Otherwise, it is sufficient to propose the 
design of an experiment for a make-or-break test, or to further advance a theory toward testable 
predictions. 

This requires that the author must identify the critical make-or-break issues for their immediate area 
of investigation. Also, the proposed next-step must be consistent with the scientific method, with due 
consideration for the current status of the topic as specified by the author. Further note that, depending on 
the status of the proposed task, independent verification may be warranted. In such a case, the vested 
interests of the Principle Investigator must be taken into account. This is a mandatory criteria, which 
means that a failure to meet this criteria (zero score) will result in a total score of zero. 

Criteria 8: Credible results 

This criteria grades the expected fidelity of the conclusions to be reached at the end of the proposed 
task. Will the task result in knowledge that will be a reliable foundation for future research decisions? 

Successful completion of the research task is defined as learning more about reaching the 
breakthrough, rather than actually achieving the breakthrough. Negative test results are considered 
progress. What is required for successful completion is that the work reaches a credible resolution that is 
clearly communicated. This criteria is judged on a combination of the realism of the proposed work, its 
cost and schedule, and on the credentials of the proposed research team and their facilities. If it is likely 
that the work can be completed within the funding and time allocations specified, and that the results will 
be accepted by other researchers as a credible foundation for future work, then a high score is warranted. 
Note too that, depending on the status of the proposed task, independent verification may be warranted. In 
such cases the vested interests of the Principle Investigator must be taken into account to ensure that there 
is no conflict of interest in the outcome of the device, phenomenon, or theory under test. This is a 
mandatory criteria, which means that a failure to meet this criteria (zero score) will result in a total score 
of zero. 

Criteria 9: Triage 

Will this research be done anyway or must this Project support it? This criteria addresses the 
possibility that the BPP Project can save its resources if the topic is likely to be explored without support 
of the BPP Project. 
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Criteria 10: Cost 

This is a reciprocal scoring factor that addresses practical resource concerns. The more costly the 
work, the lower the overall score, all other factors being equal. 

Criteria 11: Time 

This is a reciprocal scoring factor that addresses practical resource concerns. The longer to reach a 
reliable conclusion, the lower the overall score, all other factors being equal. 


Lessons Learned 

Overall, the strategy and prioritization methods of the BPP Project were found to be effective, 
although areas of improvement have been identified. The Project strategy has met with positive reviews, 
the selection process has worked well in practice, and the publication of research proves that incremental 
progress can be affordably produced on BPP topics. Some of the improvements, particularly refinements 
to the research criteria themselves, have already been incorporated. Each of these lessons are explained in 
more detail next, along with descriptions for other improvements that were being implemented at the time 
that funding for the Project was deferred. The planned improvements, which included shifting the 
operation of the Project to a nonprofit consortium of government, university and industry, were intended 
to improve the ability of the Project to respond to the large number of researchers and research 
approaches, to identify the most pressing research issues needing attention, and to provide continuity to 
better direct progress. This is discussed in the “Revisions Initiated” section. 


Strategy Lessons 

The overall BPP Project Management strategy, specifically to identify diverse near-term issues and to 
concentrate on acquiring reliable information rather than grand claims, has met with positive reviews. 
Quoting from an independent review panel of the Space Transportation Research Program, of which the 
BPP Project was a subset: 

“[The BPP] approach was unanimously judged to be well thought out, logically structured and 
carefully designed to steer clear of the undesirable fringe claims that are widespread on the Internet. The 
claim that the timing is ripe for a modest program of this sort was agreed to be justified: Clues do appear 
to be emerging within mainstream science of promising new areas of investigations. The team concurred 
that the 1997 BPP kickoff workshop did identify affordable candidate research tasks which can be 
respectably pursued, and that the proposed research prioritization criteria were a valid way to select from 
amongst these (and future) proposals. The program approach was deemed to be sound: emphasizing near- 
term progress toward long-term goals; supporting a diversity of approaches having a credible scientific 
basis; holding workshops to exchange ideas; solicit constructive criticism and assess progress; aiming 
toward testable concepts.” (Merkle 1999) 


Solicitation Process Lessons 

Regarding the utility of the solicitation process, specifically the process formally implemented over 
1999 and 2000 (NRA-99-LeRC-l), the following lessons were learned, each of which is explained in a 
subsequent paragraph: 

• Multiple reviewers per proposal is vital 

• Two-stage review process is effective 
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• Summary sheets are effective 

• Some criteria needed rearranging (revisions already incorporated) 

On topics as visionary as BPP, it is prudent to have numerous reviewers to provide well rounded 
assessments. For the 1999-2000 solicitation, at least 4 reviewers per proposal were secured for all 60 
proposals, with virtually all proposals being scored by 5 reviewers. Although it was challenging to secure 
the volunteer services of the 50+ reviewers required for this coverage, having this many reviewers 
allowed several important functions to be employed. First, it was easy to avoid conflicts-of-interest by not 
having any proposals scored by people from their own, or competing, institutions. Next, it was possible to 
match the areas of expertise of the proposals and reviewers, which varied greatly since there is a wide 
span of physics underlying the BPP ambitions. And finally, it was possible to calculate the standard 
deviations of reviewers’ scores to flag any disparate reviews for further scrutiny. This feature, of 
assessing the relative agreement amongst the reviewers of a given proposal, was found to be an effective 
tool for screening out questionable proposals and to identify problems with any reviews. 

Having a two-stage review process was found to be effective. The first stage allowed each proposal to 
receive a thorough review, while the second stage allowed a smaller team to effectively compare the 
proposals against one another. The first stage produced a large amount of data that was, by itself, not 
sufficient to identify the best proposals. Even after this data was analyzed to obtain an average score for 
each proposal and the value for the standard deviation between the reviewers’ scores, further discussion 
was required to effectively review and distill these results. For this, a smaller team of reviewers was 
assembled. This team consisted of 10 government employees from multiple government labs, including 
NASA, the Department of Energy, and the Department of Defense. With such a smaller, but still diverse, 
group of reviewers, it was relatively easy to sort through the total scores, evaluate the disparate reviews, 
and then select a diverse suite of research from top-ranking proposals. This second-stage of the process 
required about 2 1/2-days. 

The use of proposal summary sheets, like that in appendix C, was also found to be effective. These 
are the forms where the submitters encapsulate the key points of their proposal to streamline the review 
process. This made it very easy to quickly filter out noncompliant proposals and to focus on the key 
points needing further scrutiny. This might also become a useful tool for pre-proposal screening, where 
submissions of just the summary sheets could be used as a first screening, where only those submissions 
with acceptable summary sheets are invited to submit a full proposal. Also, a similar form was required 
from the reviewers, which made it easy to compile the scores. An example of a Research Increment 
Summary Form, specifically the one being drafted for a future solicitation, is presented in appendix C. 
This format is also being considered as the header for archiving the BPP research findings and 
opportunities in an electronic database. 

Regarding the criteria themselves, some revisions were needed based on the lessons learned. First, the 
Readiness Scales needed to be revised. In the original solicitation, the applicability criteria (called 
“Directness” in the NRA solicitation) and the Scientific Method readiness scale were separate. When 
submitters and reviewers were scoring readiness, they often overlooked how readiness and applicability 
were linked, leading sometimes to contradictory assessments of readiness and progress. This is why the 
revised readiness scale has these two criteria explicitly interwoven. The other problem was that the 
“Lineage” criteria was found to be redundant to the “Probability of successful completion” criteria. 
Lineage has now been deleted since it is integral to what is now called “Credible Results” (criteria 8) 
described in this report. 


Cost-Effectiveness 

In regards to whether the strategic approach of the BPP Project was cost effective, data on the cost 
and products of the Project have been compiled into table 1 so that Project Managers of other similar 
topics can compare this data to their own practices. It is difficult to directly compare such productivity to 
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other similar efforts because there are so few efforts where the data is publicly available. The one 
example for a related topic that is publicly available, in addition to the BPP Project, is from a 
congressionally mandated task to the West Virginia Institute for Software Research. Data for this single 
research task is also provided in table 1 . 

Summarizing the data in table 1, the BPP Project addressed 8 different research topics, produced 13 
journal publications on those findings, and produced numerous other publications and information 
exchange opportunities for roughly 1-1/2 million dollars, spread out over 7 years. A summary of these 
and other research tasks has been recently compiled (Millis 2004). In addition, preliminary work for 
converting the operation of the Project into a non-profit consortium was supported. 


TABLE 1— COST-EFFECTIVENESS DATA 


! NASA Breakthrough Propulsion Physics Project (1996-2002)“ 

$1,554K 

Activity 

Publications 

Item 

Cost 

Category 

Cost 


| Research 

$721K 


Macaly: Experimental and theoretical 
investigations into quantum vacuum 
energy. 

- 9 Journal Publications 

- Numerous Conference Publications 

$283K 



Malloy: Experimental and theoretical 
investigations into superluminal 
quantum tunneling. 

- 3 Journal Publications 

- Numerous Conference Publications 

$170K 



Ringermacher: Experimental test of 
EM torsion theory. 

- 1 Conference Publication 

$100K 



Cramer: Experimental and theoretical 
test of transient inertia claims. 

- 1 Conference Publication 

$49K 



Robertson: Experimental Cavendish 
balance test of RF pumped 
superconductor claims. 

- 1 Conference Publication 

$37K 



Deck: Theoretical assessment of deep 
Dirac energy levels. 

- 1 Journal Submission (in review) 

$13K 



Fralic: Experimental test of Schlicher 
thrusting antenna claims. 

- 1 Conference Publication 

$5K 



Millis: Theoretical problem definition 
of “space drives.”. 

- 1 Journal Publication 
(publication fees) 

$1K 



| Miscellaneous research support purchases 

$63K 



On-Site Contractor (3 yrs.) to handle unsolicited correspondence and Project documentation 

$176K 


Consortium initiation 

$163K 


Kick-off Workshop (1997) 

- 1 Workshop Proceedings 

$50K 


Program Support 
(overhead, averaged $63K/yr.) 

- 1 Journal Publication 

- 6 Conference Sessions 

S444K 


Congressionally Earmarked Project (2001-2001) 



$2,000K 

Institute for Scientific Research: 
Theoretical and experimental tests of 
Heaviside force for propulsion 

- 1 Journal Submission (in review) 

- 1 Conference Publication 



a Of the $1,554K funds allocated to the BPP Project, $1,354K came from the Advanced Space Transportation Program managed at Marshall 
Space Flight Center, and $200K came from the Office of Space Science, managed by NASA Headquarters. 


Areas Needing Improvement 

Overall, the strategies and review process were found to be effective, but further improvements are 
warranted. Over the course of the Project it became apparent that there are many more published research 
options than have been credibly assessed, and there has been no overall comparative assessments of these 
options to identify the most promising and cost-effective research avenues. Even though a solicitation 
process inherently includes such a review, such solicitation reviews are limited to whatever proposals are 
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received. It is likely that not all of the critical issues will be represented by quality proposals. To counter 
this situation, the following four strategies are under consideration: 

• Conduct a study to contrast the BPP challenges and emerging physics to identify research 
connections. 

• Establish an Internet-accessible comparative research database. 

• Provide better networking with the geographically dispersed physics practitioners. 

• Provide means for NASA to address those areas that are overlooked via in-house research 
capability. 

Ideally, it would be useful to have an overall assessment of the emerging physics and how it 
compares to the goals of the NASA Project. This is a daunting task as it covers much diverse ground and 
relevant questions have only begun to be systematically raised. One approach is to divide this ambition 
into the three BPP Project goals and for each goal identify the most critical physics issues needing 
resolution. The closest example of such an analysis, to date, was an assessment of the Project’s 1 st goal — 
to eliminate the need for propellant mass (Millis 1997). Similar studies on the 2 nd and 3 rd goals are 
desired. Further still, it is desired to have an overall assessment that links the critical unknowns related to 
the Project’s propulsion goals to unresolved issues of general physics. 

To allow comparison of the variety of research options, it would be prudent to establish a comparative 
database of all proposals and research suggestions received. Since the relative value of these, with respect 
to the BPP Project, could be judged by the BPP selection criteria, these criteria would serve as the logical 
fields with which to summarize, sort, and comparatively rank this information. The Research Increment 
Summary form, like that shown in appendix C, could address this essential information to allow this kind 
of ranked archiving. Further links could also be added to provide access to the details of the research 
options. The implementation of this strategy is one of the features of the nonprofit consortium discussed 
in the “Revisions Initiated” section. 

Even with such a ranked archival database, however, the information is still dependent on receiving 
quality proposals into the system. It is desirable to have a study that contrasts emerging physics to the 
goals of the BPP Project to identify the pertinent unknowns, critical issues, and curious effects that still 
require resolution. With the exception of the one paper that defined specific challenges for the first 
“Grand Challenge” of the BPP Project (Millis 1997), there have not been other assessments to identify the 
unresolved areas of physics that pertain to the BPP challenges. “Problem definition” studies for the 
second and third grand challenges are needed as well. One approach to provide the structure for 
organizing this type of complex interconnections is the “Traceability Tree” (figure 1) discussed in one of 
the prior BPP studies (Millis 1996), and further refined in a work that suggests applying the tools of fault- 
tree analysis to this challenge (Zampino 2003). 

Another area needing improvement is to provide better access to the various researchers that can add 
value to this topic. The difficulty is that such researchers are geographically scattered across the world 
and do not necessary know which BPP problems need more rigorous attention. The Internet database is 
one means to improve this feature, as are the problem definition studies. But even beyond these strategies, 
it would be prudent to simply increase the number of capable researchers that are interested in addressing 
the topic. Even though this outreach has been happening over the course of the Project, a more deliberate 
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Figure 1. — Breakthrough propulsion physics project traceability map. This figure is presented here only 
to introduce the concept of the Traceability Map as opposed to being a substantive example. It is a 
work in progress. Conceptually, this tool provides a framework against which to plot the correlations 
between emerging physics and the desired propulsion breakthroughs. In the left-most column, the 
various disciplines of physics are listed, which then branch out into specific items of interest relative 
to propulsion physics. In the right-most column, the desired goals of the project are listed, which then 
branch out into a variety of hypothetical concepts for achieving the desired effects. In-between these 
two extremes are where the relevant effects, unknowns, and issues are posted. These are linked to the 
physics and to the propulsion concepts with which they pertain. When done thoroughly, this linking 
can determine which effects, unknowns, and issues warrant the most attention to help prioritize 
research. In the interim it simply serves as a way to plot the various activities and their relations. In 
the color version, color-coding indicates the progress of each block relative to the scientific method 
readiness levels described in appendix A. 


attempt to interest more physics students in this endeavor would increase the overall progress toward 
answering the BPP Project challenges. 

Many of these improvement strategies were intended to be addressed with the implementation of a 
research consortium, which is discussed next. 


Revisions Initiated 

Traceability Map 

One of the areas needing improvement was to have a better way to identify the physics issues that 
pertain most to the goals of breakthrough propulsion. One tool being explored by NASA to address this 
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need is the “Traceability Map” shown in figure 1. This map provides a means to compare the links 
between topics in emerging physics and the critical issues related to the desired propulsion goals. By 
applying techniques similar to a fault tree analysis to such a mapping, the topics that have the most 
importance can be more easily identified (Zampino 2003). Such a framework can also serve as a map 
against which to plot known research. Specifically, if a particular item (block on the map) has been, or is 
being investigated, links to that more detailed information can be added to the block. In this way, the map 
becomes an introductory reference to grasp the range and diversity of research related to the topic. 


Research Consortium 

To more easily access the large number of geographically dispersed participants in BPP research and 
to utilize the unique advantages of government, universities, and industry, a Breakthrough Propulsion 
Physics Research Consortium is the next evolution planned for the BPP Project. In 2002, a Cooperative 
Agreement between NASA and the Ohio Aerospace Institute (OAI) was established to implement this 
consortium. OAI is a nonprofit organization whose charter involves fostering collaboration between 
government, universities, and industry on the sciences and technologies of mutual interest. 

The consortium will consist of an Advisory Council, invite membership from the large number of 
geographically dispersed participants in BPP research, will conduct research solicitations, provide 
management oversight of selected research, and maintain an electronic database of research proposals and 
results. The planned functions of the consortium are briefly described next. 

Although on hold due to the deferment of BPP funding, many of the operating methods for the 
consortium have already be set into motion. Should the NASA funding resume, or funding be secured 
from another source, the consortium can be readily reactivated. 

BPP Advisory Council. — It is envisioned that the Advisory Council would consist of several 
renowned representatives of the physics community; a representative from each of the following 
government organizations: NASA, the Department of Defense, and the Department of Energy; a 
representative for any other consortium sponsors; and an administrative, non voting, member from OAI. 
The BPP Advisory Council will manage reviews, solicitations, and selections of research tasks; manage 
the electronic database of the consortium; oversee workshops to foster information exchange on 
propulsion physics; and propose additions, replacements or terminations of members of the consortium. 
The Advisory Council’s recommendations will be formalized through a voting membership structure, as 
will decisions regarding activities funded with resources from multiple nongovernment sources. 

BPP consortium membership. — In addition to NASA as the prime sponsor for the BPP consortium, 
other groups and individuals may become participating members, subject to the recommendation and 
approval of the Advisory Council. There would be different levels of membership, including Sponsors 
and the Researcher Network. Membership can include participating universities, industries, other 
government organizations, or individual researchers. Participation may be as a sponsor or as a recipient of 
research funds. Also, a network of BPP researchers will be part of this membership. This network can be 
geographically dispersed and will consist of established researchers who can provide reviews of materials 
sent to the BPP consortium, as well as identify and/or propose candidate research. Intellectual Property 
and Nondisclosure Agreements for the members will be patterned after methods used in existing research 
consortia managed by the OAI. 

BPP consortium research solicitations. — The consortium would issue regular, formal research 
solicitations. All research submissions will be evaluated by a peer review using a two-stage review 
process similar to the first BPP research solicitation (NRA-99-LeRC-l). In the first stage, a network of 
reviewers from members of the consortium, will numerically score the proposals through the 
Consortium’s election ic submission and review database, which will use the BPP Research Prioritization 
Criteria. In the second stage, the numeric results will be compiled by the consortium to indicate the top 
ranking candidates. These results will be submitted to the Advisory Council for their review. The 
Advisory Council will select which tasks shall get consortium support and make recommendations to the 
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NASA BPP Project Manager for when NASA in-house tasks could be supported outside the consortium. 
The review process for proposals will assess credibility of the proposed research, not the likelihood that 
the areas of investigation will produce a breakthrough. Even research that fails to produce a breakthrough 
has the valuable consequence of identifying where future investments are not warranted. The Advisory 
Council is at liberty to not select any tasks in the event that no credible research proposals are received. 

BPP electronic submission and review database. — To enable the participation of geographically 
dispersed members, and as a means of providing “clearinghouse” information about BPP research, the 
BPP Research Consortium would implement an electronic database. The database will be accessible 
through the Internet, regardless of the user’ s platform. Appropriate security measures will be incorporated 
to ensure that access levels are aligned with user categories. The publicly-accessible portion of the 
database will list currently sponsored research, a bibliography of completed research articles, and allow 
submission of research proposals, suggested topics, or research results. The Researcher-Network- 
accessible portion of the database will provide access to submitted research proposals, suggested topics, 
or research results so that these can be reviewed and scored. This portion of the database will be tailored 
to fit the specific Prioritization Criteria of the BPP Project and will have automatic ranking functions 
based on submission information and review scores. The Sponsor-accessible portion of the database will 
provide access to the results of the reviews, where research options are comparatively ranked. An 
example of how this summary information might be displayed, is included in appendix C. 

BPP workshops and conferences. — When the BPP Research Consortium and its electronic 
submission/review database are established, regular BPP conference sessions can be reestablished and 
managed by the consortium. The electronic submission and review database will facilitate the peer-review 
functions needed to screen these conference paper submissions. Workshops will be convened as 
appropriate. For both workshops and conferences, proceedings or other forms of reference-able 
documentation are required. 


Resumption of In-House Research 

Another planned revision to the BPP Project is to reactivate NASA in-house research. The NASA in- 
house tasks are planned as a separate budget line item within the BPP Project to ensure that NASA has at 
least one BPP research task in addition to the tasks that are competitively selected through open 
solicitations. The candidate NASA in-house research tasks will be subject to the same peer-review 
selection process as those of the open solicitations. The BPP Research Consortium will conduct these 
reviews, and the BPP Project Manager will select the NASA in-house tasks after considering the 
recommendations from the Advisory Council. NASA tasks will be subject to the same conditions of all 
BPP research — that the research tasks are affordable, near-term, and credible research that will make 
incremental progress toward the Project’s Technical Challenges. By having a sustained in-house 
capability, NASA will have the means to address approaches not covered by the externally proposed 
research. Such an activity also helps maintain continuity and core -competency at NASA for the BPP 
topic. 


Concluding Remarks 


On topics as visionary as seeking breakthroughs in propulsion physics, it is especially challenging to 
proceed in a credible and constructive manner. From the experiences of the BPP Project over its initial 7 
years, some of the most useful strategies have been: 

• Breaking down the long-range goals into near-term immediate “go/no-go” research objectives 
that can each be assessed within 1 to 3 years. 
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• Putting the emphasis on the reliability of conclusions rather on than their implications. 

• Having reviewers concentrate on the reliability of the information to be gained from research, 
rather than on whether the concept under investigation is feasible. 

• Devising a numerical means to impartially compare research options and readily reject “fringe” 
submissions. 

• Addressing a diversified portfolio of research approaches. 

It is expected that many of these strategies could be adapted to other leading-edge research projects 
that face similar challenges. 
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Appendix A 

Applied Science Readiness Levels 

Using the Science Method as a Research Management Tool 


The following readiness levels were devised to provide a ranking system for applied science in an 
analogous manner to how the Technology Readiness Levels (Hord 1985) are used to rank relative 
maturity of engineering developments. Specifically, these Applied Science Readiness Levels (SRLs) 
consist of 3 stages for applicability (reflecting how research evolves from the more general understanding 
to the more specific applications), and within each of these three levels, 5 steps of the Scientific Method 
are repeated. This equates to 15 levels of relative maturity, with the most advanced level being equivalent 
to Technology Readiness Level 1 (basic principles observed and reported). 


APPLIED SCIENCE READINESS LEVELS 


SRL-1.0 

General Physics — Prescience (unconfirmed effect or new information connection) 

SRL-1.1 

General Physics — Problem formulated 

SRL-1.2 

General Physics — Data collected 

SRL-1.3 

General Physics — Hypothesis proposed 

SRL-1.4 

General Physics — Hypothesis tested and results reported 

SRL-2.0 

Critical Issues — Prescience (unconfirmed effect or new information connection) 

SRL-2.1 

Critical Issues — Problem formulated 

SRL-2.2 

Critical Issues — Data collected 

SRL-2.3 

Critical Issues — Hypothesis proposed 

SRL-2.4 

Critical Issues — Hypothesis tested and results reported 

SRL-3.0 

Desired Effect — Prescience (unconfirmed effect or new information connection) 

SRL-3.1 

Desired Effect — Problem formulated 

SRL-3.2 

Desired Effect — Data collected 

SRL-3.3 

Desired Effect — Hypothesis proposed 

SRL-3.4 

Desired Effect — Hypothesis empirically tested and results reported (equivalent to TRL 1: Basic 
principles observed and reported) 


Three stages of applicability . — The scope of scientific research can span from the very general, 
broad-sweeping considerations all the way down to specific details of a given application. The more 
focused that a research increment is toward a desired application, the greater its applicability level, as 
reflected in the following 3 stages: 


Stage 1 : General physics 

The research topic deals with general underlying physics related to the desired application. 


Stage 2: Critical issues 

The research topic deals with an immediate unknown, critical make or break issue, or curious 
effect relevant to the desired application. 


Stage 3: Desired effects 

The research topic deals with a specific effect or device for answering the goals of the 
application. 

Scientific method . — Within each of these ranges of focus, the scientific method can gauge progress. 
The following definitions for the steps of the scientific method have been tailored to reflect applied 
research progress. Since applied research implies a tangible product, these steps distinguish between 
empirical approaches (those based on the emergence of empirical effects) and theoretical approaches 
(those based on theory). The most significant distinction is with the final stage, where the hypotheses are 
tested. For applied science, again emphasizing a tangible product, this final stage can only be satisfied 
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with an empirical test. Another noteworthy distinction from the common definitions of the Scientific 
Method is the inclusion of the zeroth step. This gives a placeholder for emerging opportunities that have 
not yet been addressed. 

Step 0: Prescience 

Empirical'. Observations of an unconfirmed anomalous effect have been reported (includes 
observations of natural phenomena or claims of unverified devices), or 
Theory'. A correlation between a desired goal (or unsolved problem) and the existing knowledge 
base has been articulated. 

Step 1: Problem formulated 

Empirical'. An experiment has been defined that can collect the data required to isolate and 
characterize the anomalous effect, or 

Theory'. A goal (or problem) has been defined specifically enough to identify the specific 
remaining knowledge gaps toward achieving the goal (or solving the problem). 

Step 2: Data collected 

Empirical: Data has been collected and analyzed from experiment to isolate and characterize the 
anomalous effect, or 

Theory: The relevant data to fill the critical knowledge gaps, identified in the previous step, have 
been collected through experiment, observation, or mathematical proof (this level 
includes assessments of theory using mathematical analysis). 

Step 3: Hypothesis proposed 

Empirical: A mathematical representation of the physical principles underlying an effect has been 
offered to explain the effect and predict additional (testable) effects, or 
Theory: A mathematical representation of the relation between physical phenomena has been 
offered that addresses the goal (or problem) formulated previously. 

Step 4: Hypothesis tested and results reported 

The hypothesis has been tested by comparison to observable phenomena or by experiment 
sufficiently to determine if it appears viable, and the results reported. NOTE: In the context of 
applied research, testing of a hypothesis must be empirical; that means it must be done by 
comparison to observable phenomena or by experiment, rather than just by mathematical proof. 
Although mathematical proof can be used to test the consistency of a theory against the known 
science, such a mathematical test alone is not sufficient to warrant achievement of Step 4. Instead, 
a mathematical test of a theory reflects achieving Step 2. 

Once a research objective has been defined and its maturity identified according to the 
Applied Science Readiness Levels, the next logical increment of research would be to advance 
the topic the next level of the Applied Science Readiness Levels. 
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Appendix B 

Basic Research Evaluation System 


Bruce Banks 

National Aeronautics and Space Administration 
Glenn Research Center 
Cleveland, Ohio 44135 


This appendix summarizes the decision-making process devised by Bruce Banks of the NASA Glenn 
Research Center that was used as a foundation for the research prioritization process of the Breakthrough 
Propulsion Physics Project. It is designed for the prioritization of plans where there are many issues of 
varying importance and degree of influence that tend to make such selections complex. A multiplicative 
scoring method is employed, based on principles of probability and statistics, that is significantly more 
sensitive than additive scoring methods. 

This procedure starts by assembling a team of representative experts and customers of the desired 
technology. Through brainstorming and voting, the team defines the relevant evaluation criteria, and then 
narrows these criteria down to a minimal list with weighting factors for each. The group also must 
distinguish between those criteria that are mandatory (criteria that must be met), and those criteria that are 
just enhancing. It is essential that the customers for the research concur with the criteria and it is crucial 
that the other participants concur before applying the criteria to actually evaluate the options. 

A recent example of an application of this process is the selection of the replacement thermal control 
materials for the Hubble Space Telescope (Townsend, J.A., Hansen, P.A., McClendon, M.W., de Groh, 
K.K., and Banks, B.A. (1999), “Ground-based testing of replacement thermal control materials for the 
Hubble Space Telescope,” In High Performance Polymers, vol. 11, pp. 63-79). 

The process has the following advantages and characteristics: 

• It is ideal for making the best decision when many complex and often conflicting issues must be 
considered. 

• The process is applicable to prioritizing choices and making decisions on almost any topic such 
as: 

Strategic planning (determining which of several potential plans is best) 

Personnel selections or promotions 
Career options 
Contractor selections 
Major purchases 

• Every issue and every opinion is considered. 

• One individual or a team can perform the process, but the team approach is recommended on 
those decisions where a team consensus on the final decision is paramount. 

• The decision-making process results in excellent “buy-in” by those using the process, because all 
issues and everyone’s opinions can be taken into account. 

• The process employs multiplicative scoring that is significantly more sensitive to critical issues 
than additive scoring methods. 

• The decisions are prioritized in a quantified manner. 

• The resulting decisions are highly defendable. 

• The process minimizes debate or skewing from overly-assertive people. 

• The process can be performed remotely, without an actual team meeting, by simple, electronic 
transference of information files via e-mail. 

• The mathematical methods of the process can be automated in software. 
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Evaluation Criteria 


To determine the criteria to be used in the selection it is helpful to have a suite of illustrative 
options — a sample of items whose prioritization is sought. With these up for discussion, it is easier to start 
collecting a list of evaluation factors. When the committee discusses the evaluation criteria, every 
proposed criteria should be listed, regardless of how many people feel it is meritorious, because after all 
criteria are listed, individual or group voting will be done to determine the relative importance of each. 

Evaluation criteria should be: 

• Phrased in a positive sense (express as desired characteristics or freedom from undesirable 
characteristics) 

• Phrased such that there is majority acceptance of the wording 

• Independent of each other (no duplication of issues) 

• Address single issues (issues should not contain the word “and”) 

• Able to be numerically scored (or graded) 

• Include all relevant issues 


Relative Importance of Criteria 

Once the criteria have been selected, the committee decides, by consensus or majority vote, what the 
most important criteria is. The “relative importance” of this criteria is assigned a value of 1. (Comment: 
The committee can decide that this is a fixed maximum and that no other criteria can be given a value 
greater than 1 by any committee member. This would prevent “skewing of the system” by any one 
individual.) 

Voters decide their values for relative importance of each evaluation criteria knowing that: 

• The most important criteria is rated 1, and is not changeable. 

• Because the most important criteria has been rated 1, the relative importance of the remaining 
criteria will have proportional importance values between 0 and 1 . For example, a criteria half as 
important as the most important criteria should be rated 0.5 

• Values can be entered which have up to three decimal places to the right of 0. 


Scholastic Grading 

Experience has shown that an evaluation depends not only on the perceived merit of the idea, but also 
on the evaluators’ interpretations of how to score the idea. For example, if the scoring range is 0 to 25 on 
a given criteria, such as with the Small Business Innovative Research (SBIR) evaluations, two different 
evaluators may use significantly different point values to mean the same grade. To avoid this problem, it 
is recommended to use a familiar and limited grading system such as the scholastic 4-point scale: 


A (4 points) 
B (3 points) 
C (2 points) 
D ( 1 point) 
F (0 points) 


Excellent or outstanding, meeting the criteria to the maximum amount. 
Good, or well above average. 

Average, or the score to use if there is no reason to score high or low. 
Poor or well below average. 

Fails to meet the criteria. 


In those cases where these discriminators do not fit, it is still recommended to have the scoring range 
limited to about 5 gradations where possible, and to have clear text explanations to accompany each 
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gradation. Since the final scores combine several criteria, it is possible to get sufficient distinctions with 
the total final scores even with such limited gradations. 


Span of Influence of Criteria 

Each prioritization criteria is able to influence the overall score, from a possible maximum of 
outstanding to some lower level, depending upon how critical the criteria is. This includes identifying 
which criteria are mandatory, meaning those for which a zero-score would eliminate the option from 
consideration. For each criteria, the committee members determine the overall prioritization score that 
would result if a option were rated a perfect “4” with respect to every other criteria, but rated “0” for the 
specific criteria being considered. For example, when choosing to buy a car, the trim color could 
contribute to the overall desirability, but even if it were the worst color possible (score “0” for that criteria 
on that car), an overall good “3” rating would be appropriate for the whole car if all other criteria were 
outstanding (grade = “4”). In this example, the criteria identified as trim color would receive a span of 
influence value of “3.” 


Composite Scoring Equations 

After this step, the data can be tabulated and calculations performed to determine the normalizing 
coefficients for the composite scoring equation. 


RELATIVE IMPORTANCE, SPAN OF INFLUENCE, AND NORMALIZING COEFFICIENTS 


Criteria 

Scores 

Relative Importance 
(score between 0 and 1) 

Span of Influence 
(score on a scale of 0 to 4) 

Normalizing Coefficient 
for Score 

Normalizing Coefficient 
for Span of Influence 

A1 

R1 

SI 

kl 

cl 

A2 

R2 

S2 

k2 

c2 

A3 

R3 

S3 

k2 

c3 


The normalizing coefficients, kn and cn, mathematically scale scores for each criteria to assure the 
span of influence is properly accounted for within the 0 to 4 scoring scale. Or stated differently, the 
coefficients are set so that the parenthetical terms of the composite scoring equation have a maximum 
value equal to 4. 

For this example, the basic composite scoring equation is as follows: 


Composite Score(A) = (k\-A\ + cl • Si)* 1 (k2 -A2 + c2- S 2)* 2 (k3 • A3 + c3 • S3)* 3 


Where A represents the option being scored, An reflects the score given for the n h criteria on option 
A, and the other variables are as shown in the table above. 

The mathematics to determine the normalizing coefficients can be automated in software. 


Scoring of Candidates 

Once the relative importance, span of influence, and normalizing coefficients are determined for each 
issue, then the candidates can be scored with respect to each of the criteria using the scholastic 4-point 
scale cited previously. 
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Appendix C 

Research Increment Summary Form 


REVIEW COLUMN 


B: Title of submission (for published results, list full reference citation here) 


D: Central issue, unknown, or observation understudy (paragraph) 


E: Findings [Only required when citing published results] (paragraph & check box F) 


G: Relevance to Project (paragraph & check-boxes H) 


I: Increment of work proposed or reported (paragraph & check boxes J,K,L,P,S) 


M: Contrasting or skeptical challenges to the proposed (or reported) approach (paragraph) 


N: Founding References (list) 


Q: Contrasting or Skeptical References (list) 



R: Prior publications to reflect proposed researchers' qualifications (list) 


T: Performing Organization 


U: Other Sponsors (optional) 


Submission ID#: 


C: Submission Type: 

^Published Results 
(blocks R,S optional)) 

Proposal 
(Skip blocks E,F) 

[^Suggested Inquiry 
(Fill in at least A-D,G,K,N) 


F: Sequels Expected? 

□ □ □ 

No Maybe Yes 


H: Ultimate Improvement 

Mass mm 

0 12 3 4 
Speed I I I I I I 
0 12 3 4 
Energy I I I I I I 

0 12 3 4 


RELEVANCE 

1: Gains toward Goals? 

-nun □ 

0 1 2 3 4 0-4 

s irrm □ 

0 1 2 3 4 0-4 

E imri □ 

0 1 2 3 4 0-4 



KL: Readiness & Progress 
K: Befor< L: After I 


3-4: Readiness & Progres 
Before After 


cMDefn M 
^[JPreSci [J 1 

Science Readiness Levels 


£ Defn 
^MPreSci 


P: Related Disciplines I 
(Check all that apply) | 

_ Basic Motion & Energy 
_ Electromagnetism 
_ Special Relativity 
Quantum Mechanics 
_ QM: Vacuum 
_ QM: Non-Locality 
_ QM: Gravity 
_ Particle Physics 
_ Micro/Nano Physics 
_ Cryogenic Physics 
Superconductors 
_ Cosmology/Astron 
_ General Relativity 
_ GR: Metric Engr 
_ GR: GravitoMagnetic 
_ Strings/Branes J 


(4) Calculated Progres: 

mm □ i 


CREDIBILITY 
5: Foundations 

mm nn 

F D C B A 0-4 
6: Contrasts 

mm nn 

F D C B A 0-4 
7: Testability 

mm nn 

F D C B A 0-4 
8: Results 

nrrm □ 

F D C B A 0-4 


Triage: 

□ □ □ 

Likely other ? BPP uniqu 


Cost: $| |K 

Time: 

□ □ □ □ 

=3y 2y ly <ly 


9: Triage: 

HD CD 

D C A 1-4 

10: Cost 

nrrm nn 

F D C B A 0-4 


1: Ti me (yrs) 

rfm \zn 

























Appendix D 

Research Increment Evaluation Instructions 


The following text more thoroughly describes each prioritization criteria, including the scoring 
gradations. This text is presented in the context of reviewer instructions, to more clearly convey how 
these criteria would be employed during a review. The descriptions that follow also identify which criteria 
are mandatory and which are not. 


Criteria 1 — Gain on Goals 

This criteria grades how the proposal relates to all three of the BPP Grand Challenges, assuming that 
the concept behind the proposed increment of research ultimately reaches fruition. Each Grand Challenge 
is graded separately and the final Gain criteria involves the sum of these three subset scores. 

It is mandatory that the proposed work seek advances in science that are in some way relevant to the 
Project’s three propulsion challenges or any critical issues or unknowns related to these goals. The scope 
is limited to further advances in science from which genuinely new technology can eventually emerge — 
technology to suipass the limits of existing methods — as opposed to further developments of known 
technology. This means that if the proposed work only builds on known technology, then it fails this 
criteria. 

For each of the three Grand Challenges, specify which of the statements best describes the ultimate 
achievable performance of the concept being addressed by the proposal, while entertaining the 
assumption that a final embodiment ultimately functions as desired. Granted, it may be difficult to predict 
this ultimate impact since the concepts may be far from fruition or the concept may appear not to be 
viable. For grading this criteria, assume for the moment, that the concept is viable and will reach fruition. 
Other criteria will grade readiness and credibility. 

Note that this is a mandatory criteria, which means that a failure to meet this criteria (a zero score on 
all three subcriteria) will result in a total score of zero. Since the scores for all three Grand Challenges 
will be added, it is only mandatory that one of the three goals be addressed. 

BPP Grand Challenge 1: Mass. — Discover new propulsion physics that eliminates or dramatically 
reduces the need for propellant. 

Scoring gradations'. 

0 Not applicable to this goal (default answer if no answer specified). 

2 Applicable, but potential impact unknown. 

3 Intended to significantly reduce propellant requirement. 

4 Intended to eliminate the need for propellant and the need for directed energy. 

(The term, “directed energy” means any form of energy sent from a central location such as 
from the Earth or Sun.) 

BPP Grand Challenge 2: Speed. — Discover how to circumvent existing speed limits to dramatically 
reduce transit times. 

Scoring gradations'. 

0 Not applicable to this goal (default answer if no answer specified). 

2 Applicable, but potential impact unknown. 

3 Intended to eliminate speed constraints caused by limits of propellant or energy supply. 

4 Intended to circumvent the light speed limit. 
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BPP Technical Challenge 3: Energy. — Discover new energy physics to power these propulsion 
devices at levels sufficient for interstellar flight. 

Scoring gradations'. 

0 Not applicable to this goal (default answer if no answer specified). 

2 Applicable, but potential impact unknown. 

3 Better energy conversion physics, but still limited to a consumable onboard supply. 

4 Intended to provide energy sources and conversion methods accessible in flight. 


Criteria 2 — Empiricism 

Does the topic deal with tangible physical effects or just theory? Since this NASA Project is 
interested in advancements that can eventually lead to new technology, and since empiricism is necessary 
to validate theories, there is a decided preference toward empirical observations over purely analytical 
studies — all other factors being equal. Experiments, being hardware, are considered closer than theory to 
becoming technology. Also, experiments are considered a more direct indicator of how nature works. 
Theories are interpretations to explain observations of nature, while the empirical data is nature. The most 
desired research task is an experiment that is coupled with theory. Experiments which are backed by a 
sound theoretical foundation provide a means to numerically assess the utility and scalability of the 
effects beyond just a single demonstration experiment. The next preference is experimental work by 
itself; for example, to independently test a claimed anomalous effect. The next preference is theoretical 
work by itself. Lowest on this priority scale is work that only involves comparative studies of existing 
approaches or literature searches. 

Scoring gradations'. 

1 Comparative study, data collection, or literature search. 

2 Theoretical work only, without empirical investigations. 

3 Experimental tests or empirical observations only. 

4 Experiment or empirical observations coupled with theory. 


Criteria 3 — Readiness 

This criteria measures the present maturity of the topic/concept under study, as measured against the 
Applied Science Readiness Levels. This criteria and criteria 4 are answered jointly in the BPP Readiness 
Level Scoring Gradation Table, shown below. The specific value entered for this criteria is the score (left 
column) corresponding to the appropriate readiness level (right column). 


Criteria 4 — Progress 

This criteria measures the anticipated progress to be achieved by the research task, as measured by the 
numeric difference between the before and after readiness scores. The before level is simply criteria 3, 
Readiness. The after is the readiness level expected to be achieved upon completion of the research task. 
Both of these use the numeric scores associated with Applied Science Readiness Levels, shown in the 
Progress Scoring Gradation Table below. 

Scoring gradations for criteria 3 (readiness) and criteria 4 (progress). — Using the BPP Applied 
Science Readiness Level Table below, specify the status to be achieved upon completion of the proposed 
task (“after”), and its status today (“before”) for the work being proposed. Answer this question within the 
limits of the specific incremen t of research being addressed by the proposal. Note that “successful 
completion” is defined as completing the proposed work and learning more about reaching the 
breakthrough, rather than actually achieving the breakthrough. Negative test results are still progress. 
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Important note: Be sure that the score value assigned for after is greater than or equal to the score value 
for before, and furthermore, note that an equal value for both before and after means zero progress, which 
will fail the proposal. Only score this way if, indeed, the proposal is not offering any real progress. 
Appendix A contains a full description of the meaning of these readiness levels. 


BPP READINESS LEVEL SCORING GRADATION TABLE 


Score 

Score 

BPP Applied Science Readiness Levels 

Before 

(Readiness) 

After 


16 

16 

> TRL-2: Application conceptual design formulated (or higher readiness level) 



Applicability Stage 3 — Desired Effect: Deals with a specific effect or device for producing net 
force, motion, or energy-exchange that addresses the BPP Grand Challenges 

15 

15 

SRL-3.4 Desired Effect — Hypothesis tested empirically and results reported (equivalent to 
TRL-1 Basic principles observed and reported) 

14 

14 

SRL-3.3 Desired Effect — Hypothesis proposed 

13 

13 

SRL-3.2 Desired Effect — Data collected 

12 

12 

SRL-3.1 Desired Effect — Problem defined 

11 

11 

SRL-3.0 Desired Effect — Prescience (unconfirmed effect or new information connected to 
application) 



Applicability Stage 2 — Critical Issue: Deals with a critical unknown, make or break issue, or 
curious effect directly related to the BPP Grand Challenges 

10 

10 

SRL-2.4 Critical Issue — Hypothesis tested empirically and results reported 

9 

9 

SRL-2.3 Critical Issue — Hypothesis proposed 

8 

8 

SRL-2.2 Critical Issue — Data collected 

7 

7 

SRL-2.1 Critical Issue — Problem defined 

6 

6 

SRL-2.0 Critical Issue — Prescience (unconfirmed effect or new information connected to 
application) 



Applicability Stage 1 — General Physics: Deals with general physics pertinent to the BPP Grand 
Challenges in some manner 

5 

5 

SRL-1.4 General Physics — Hypothesis tested empirically and results reported 

4 

4 

SRL-1.3 General Physics — Hypothesis proposed 

3 

3 

SRL-1.2 General Physics — Data collected 

2 

2 

SRL-1.1 General Physics — Problem defined 

1 

1 

SRL-1.0 General Physics — Prescience (unconfirmed effect or new information connected to 
application) 


Criteria 5 — Credible Foundations 

This criteria grades how well the proposed work is grounded in credible foundations. The proposed 
work must be based in some way on data or theories that are in the peer-reviewed literature. Note: 
requiring reference citations is one of the techniques to filter out “fringe” submissions. This is a 
mandatory criteria, which means that a failure to meet this criteria (zero score) will result in a total score 
of zero. Grade this criteria on how well the author identifies the most relevant references for their topic of 
investigation. A variety of specific guidelines are provided below on how this criteria maps to different 
proposed situations. 

New, unreported effect. — In cases where an unconfirmed anomalous effect is being investigated 
(where the effect has not yet been independently reported nor confirmed in the peer-reviewed literature), 
the author must cite peer-reviewed references to indicate why the newly observed phenomenon would be 
considered anomalous. For example, in the case of anomalous thrust observations, where no reaction 
mass is readily apparent, it would be appropriate to cite references on momentum conservation. Note: 
requiring an admission, that the effect does not match the physics known to date, is one of the techniques 
to filter out “fringe” submissions. If the author assumes that the effect is genuine, despite not having been 
independently confirmed, or despite the appearance of contradicting known physics, then the author fails 
this criteria. For consistency in this case, it is expected that the readiness level specified under criteria 3, 
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should match “Scientific Method Step 0 (prescience),” at whatever “Applicability Stage” fits the proposed 
work. 

Known, unconfirmed effect . — In cases where an unconfirmed anomalous effect is being investigated, 
that has been reported in the literature, then these references must be cited. It is expected, in this case, that 
the existing literature would already list suspect causes for a “false positive” to help guide independent 
confirmation or refutation. For consistency in this case, it is expected that the readiness level specified 
under criteria 3, should match “Scientific Method Step 1 (problem formulated),” at whatever 
“Applicability Stage” fits the proposed work. If the suspect causes have not yet been articulated in the 
literature, then the readiness level specified under criteria 3, should match “Scientific Method Step 0 
(prescience).” If the author assumes that the effect is genuine, despite not having been independently 
confirmed, or despite the appearance of contradicting known physics, then the author fails this criteria. 

Known, confirmed effect . — In the case where the proposed work builds on an effect that has already 
been confirmed in the peer-reviewed literature, then the author must cite those references. For consistency 
in this case, it is expected that the readiness level specified under criteria 3, should match “Scientific 
Method Step 2 (data collected),” at whatever “Applicability Stage” fits the proposed work. 

New theory . — In the case where work involves a theory that is not yet in the peer-reviewed literature, 
then the author must cite peer-reviewed references of the data or phenomena with which they are claiming 
consistency. It is not necessary that the author agree with current interpretations of this data, but it is 
mandatory that the theories are consistent with credible empirical evidence. For consistency in this case, it 
is expected that the readiness level specified under criteria 3, should match “Scientific Method Step 2 
(data collected),” at whatever “Applicability Stage” fits the proposed work. 

Known theory . — In the case where the proposed work builds on a theory that is already in the peer- 
reviewed literature, then the author must cite those references. For consistency in this case, it is expected 
that the readiness level specified under criteria 3, should match “Scientific Method Step 3 (hypothesis 
proposed),” at whatever “Applicability Stage” fits the proposed work. 

Scoring gradations'. 

0 F Fails to meet. 

1 D Poor or well below average. 

2 C Average. 

3 B Good or well above average. 

4 A Excellent or outstanding, meeting the criteria to the maximum amount. 


Criteria 6 — Credible Contrasts 

This criteria grades how well the authors articulate how their proposed work compares to existing 
credible interpretations, relative to the BPP Grand Challenges. This is to ensure that an idea is oriented 
toward the goals of the Project, and to ensure that the authors have done their homework on the existing 
literature. This not only checks for relevance to BPP, but also positions the work to address the next 
criteria of a discriminating test. Note: requiring reference citations is one of the techniques to filter out 
“fringe” submissions. This is a mandatory criteria, which means that a failure to meet this criteria (zero 
score) will result in a total score of zero. Grade this criteria by how well the author identifies the most 
relevant literature and on their understanding of this literature. 

Also, recall from the objectives of the BPP Project, that the perceived correctness of the author’s 
alternative interpretations is not being judged with this criteria. Unless there is some obvious error, it is 
considered too difficult to reliably determine such feasibility during a proposal review. Such an 
assessment would constitute a full research task unto itself. The burden of addressing feasibility, via a 
discriminating test, is addressed by criteria 7. Instead, judge how well the author demonstrates an 
understanding of the current, credible interpretations that are cited, and the author’s ability to contrast this 
prior knowledge to the approach they are offering. Recall that the BPP Project examines emerging 
physics in the context of propulsion and power, and as such, there is latitude to consider alternative 
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perspectives beyond that from general physics. Even though the current credible interpretations have 
already passed their own rigorous tests, this does not imply that such interpretations are a complete or best 
representation of the actual underlying physics, especially in the context of propulsion and power. 
Conversely, however, if the proposed interpretation has already been raised and dismissed in the open 
literature, then the author must cite and address these references. 

A variety of specific guidelines are provided below on how this criteria maps to different proposed 
situations. 

New, unreported effect. — In cases where an unconfirmed anomalous effect is being investigated, 
where the effect has not yet been reported nor confirmed in the peer-reviewed literature, then the author 
must focus on comparing the effect with other, credibly known effects that might lead to a false -positive 
conclusion. In the prior criteria (“foundations”), the author had to acknowledge that the effect was 
anomalous. In this criteria, the author must demonstrate that they are astute to the conventional 
interpretations that must be tested to determine if the effect is genuinely new. References that cover these 
conventional interpretations must be cited. Also, the author must explain why the effect (if genuine) 
might be advantageous to the BPP challenges. If the author does not address the issue of ruling out the 
suspect causes, then the author fails this criteria. 

Known, unconfirmed effect. — In cases where an unconfirmed anomalous effect is being investigated, 
that has already been reported in the literature, two different scenarios can apply. If the existing published 
report (that the author had to cite under “foundations”) did not list a well-rounded set of suspect causes 
for a “false positive”, then judge adherence to this criteria in the same manner stated above for the case of 
new unreported effects. On the other hand, if the existing published report did sufficiently list suspect 
causes for a “false positive,” then the original report will suffice for the required “contrast” citation, but 
the author must still demonstrate that they are astute to the conventional interpretations necessary to 
determine if the effect is genuinely new. Also, the author must explain why their proposed investigation is 
more applicable than the existing or past investigations into the effect. In the case where null results were 
previously published, the author must cite these and explain why these prior tests were incomplete, in 
error, or why a reinvestigation is warranted. Reference citations for these other investigations are 
required. Also, the proposal must explain why the effect (if genuine) might be advantageous to the BPP 
challenges. If the author does not address the issue of ruling out the known, suspect causes, then the 
author fails this criteria. 

Known, confirmed effect. — In the case where the proposed work builds on an effect that has already 
been confirmed in the peer-reviewed literature, then the author must explain why the effect might be 
relevant or advantageous to the propulsion challenges and why their investigation is more applicable to 
BPP than the prior or ongoing investigations into the effect. Reference citations for the confirmation 
publication, and for any prior or ongoing investigations, are required. If the author is challenging the 
current interpretations of the effect, then also judge this criteria by the guidance offered under “new 
theory,” below. 

New theory. — In the case where a theory is proposed that is not yet in the peer-reviewed literature, 
then it is mandatory that the new theories be compared to the contemporary theories that address the same 
phenomena. Reference citations for the contemporary theories are required. The comparison must explain 
why the new theory would be more advantageous to the propulsion challenges than the contemporary 
theories. Judge this criteria by how well the author demonstrates an understanding of the existing theories, 
and on the author’s ability to identify the unresolved issues of both theories with respect to the goals of 
breakthrough propulsion or power. The author must also demonstrate a willingness to consider that their 
theory might be in error, by identifying its weak points. If the author assumes that their theory is correct, 
without it having been confirmed with rigorous empirical tests, then the author fails this criteria. 

Known theory. — In the case where the proposed work builds on a theory that is already in the peer- 
reviewed literature, then the author must describe how their work will be more applicable to BPP than the 
prior or ongoing work on the same theory. Reference citations for the contemporary theories are required. 
If the theory is still under debate in the open literature, then the author must acknowledge its potential 
weaknesses, and cite references that highlight these issues. Judge this criteria by how well the author 
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demonstrates an understanding of the known theory, its debated issues, and on the author’s ability to 
identify how the theory applies to the goals of breakthrough propulsion or power. 

Scoring gradations-. 

0 F Fails to meet. 

1 D Poor or well below average. 

2 C Average. 

3 B Good or well above average. 

4 A Excellent or outstanding, meeting the criteria to the maximum amount. 


Criteria 7 — Credible Tests 

This criteria judges how well the research advances the topic toward a discriminating test. It is 
required that the proposed work be leading toward a discriminating test or actually be a discriminating 
test. If a discriminating test can be completed within the budget and time guidelines requested of 
proposals, it is necessary that the test actually be proposed. Otherwise, it is sufficient to propose the 
design of an experiment for a make-or-break test, or to further advance a theory toward testable 
predictions. 

This requires that the author must identify the critical make-or-break issues for their immediate area 
of investigation. Also, the proposed next-step must be consistent with the scientific method, with due 
consideration for the current status of the topic as specified by the author. Further note that, depending on 
the status of the proposed task, independent verification may be warranted. In such a case, the vested 
interests of the Principle Investigator must be taken into account. This is a mandatory criteria, which 
means that a failure to meet this criteria (zero score) will result in a total score of zero. A variety of 
specific guidelines are provided below on how this criteria maps to different proposed situations. 

Unconfirmed effect ( reported or not ). — In cases where an unconfirmed anomalous effect is being 
investigated, a discriminating test must be suggested that could distinguish between possible conventional 
explanations or whether this is a genuine new effect. The task should propose to at least design a 
discriminating experiment, or to actually conduct an experimental test. The work will be considered more 
credible if the proposal concentrates only on the experimental methods rather than on speculating on a 
new cause for the effect. For consistency in this case, it is expected that the completion readiness level 
(the level anticipated after the task is completed), will be at least at “Scientific Method Step 1 (problem 
formulated)” for an experiment design, or “Scientific Method Step 2 (data collected)” if an experimental 
test is actually planned. 

Known, confirmed effect . — In the case where the proposed work builds on an effect that has already 
been confirmed in the peer-reviewed literature, a logical next step would be to develop a theory to 
describe the anomaly. It would also be appropriate to propose a reconfiguration of the effect so that its 
propulsive or energy implications could be assessed. For consistency in this case, it is expected that the 
completion readiness level could be anywhere between “Scientific Method Step 2 (data collected)” 
through “Scientific Method Step 4 (hypothesis tested),” depending on the breadth of the proposed work. 

Theory . — In the case where the proposed work deals with theory, it is mandatory that the new 
theories are at least matured to the point where mathematical models are offered (this is one of the 
“fringe” filters). Then, either further mathematical analysis, to predict testable effects; comparison to 
credible empirical observations; or experimental tests must be proposed that can bring the theory closer to 
a correctness resolution. An actual empirical test is preferred. For consistency in this case, it is expected 
that the completion readiness level could be anywhere between “Scientific Method Step 1 (problem 
formulated)” through “Scientific Method Step 4 (hypothesis tested),” depending on the breadth of the 
proposed work 


NAS A/TM— 2004-2 1 3406 


34 



Scoring gradations: 

0 F Fails to meet. 

1 D Poor or well below average. 

2 C Average. 

3 B Good or well above average. 

4 A Excellent or outstanding, meeting the criteria to the maximum amount. 


Criteria 8 — Credible Results 

This criteria grades the expected fidelity of the conclusions to be reached at the end of the proposed 
task. Will the task result in knowledge that will be a reliable foundation for future research decisions? 

Successful completion of the research task is defined as learning more about reaching the 
breakthrough, rather than actually achieving the breakthrough. Negative test results are considered 
progress. What is required, for successful completion, is that the work reaches a credible resolution that is 
clearly communicated. If it is likely that the work can be completed within the funding and time 
allocations specified, and that the results will be accepted by other researchers as a credible foundation for 
future work, then a high score is warranted. Base this assessment on a combination of the realism of the 
proposed work, its cost and schedule, and on the credentials of the proposed research team and their 
facilities. If cost-sharing is mentioned in the proposal, judge this criteria on the total resources to be 
devoted, not just the amount to be charged to NASA. Consider the clarity and quality of the proposal and 
any prior publications from the authors as a good reflection of the clarity and quality of the final product. 
Note too that, depending on the status of the proposed task, independent verification may be warranted. In 
such cases the vested interests of the Principle Investigator must be taken into account to ensure that there 
is no conflict of interest in the outcome of the device, phenomenon, or theory under test. This is a 
mandatory criteria, which means that a failure to meet this criteria (zero score) will result in a total score 
of zero. 

Scoring gradations: 

0 F Fails to meet. 

1 D Poor or well below average. 

2 C Average. 

3 B Good or well above average. 

4 A Excellent or outstanding, meeting the criteria to the maximum amount. 


Criteria 9 — Triage 

Will this research be done anyway or must this Project support it? This criteria addresses the 
possibility that the BPP Project can save its resources if the topic is likely to be explored without support 
of the BPP Project. 

Specify which statement best describes the situation. Note that this is not a mandatory criteria. A 
minimum score here will only result in demoting an overall “A” grade to a “C” grade. 

Scoring gradations: 

1 (D) Certain to be credibly done without the support of the BPP Project. 

2 (C) Unknown. 

4 (A) Exclusively suited to the BPP Project. 
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Criteria 10 — Cost 


This is a reciprocal scoring factor that addresses practical resource concerns. The more costly the 
work, the lower the overall score, all other factors being equal. 

Scoring gradations'. 

0 (F) If the cost is outrageous, then assign a failing grade. 

1 (D) Cost > $400K — Below average. 

2 (C) Cost = $200K — Average. 

3 (B ) Cost = $100K — Good or well above average. 

4 (A) Cost < $50K — Excellent (but verify that this is realistic for the work offered). 


Criteria 11 — Time 

This is a reciprocal scoring factor that addresses practical resource concerns. The longer to reach a 
reliable conclusion, the lower the overall score, all other factors being equal. 

Scoring gradations'. 

1 (D) Duration > 3 years — Below average. 

2 (C) Duration = 2 years — Average. 

3 (B ) Duration = 1.5 years — Good or well above average. 

4 (A) Duration < 1 year — Excellent (but verify that this is realistic for the work offered). 
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